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1.  Introduction 


1.1.  The  purpose  of  this  research 

In  what  follows  we  will  consider  the  following 

Problem: 

Given  a  sample,  determine  the  random  field  that  generated  it. 

At  first  glance,  this  problem  seems  to  be  without  solution,  because  of  the  lack  of 
sufficient  data.  In  order  to  make  the  problem  reasonable,  it  is  necessary  to 
assume  that  the  field  is  not  arbitrary  but  belongs  to  some  specific  class,  e.g., 

1)  is  composed  of  independent  random  variables 

2)  is  first  order  Markov  (e.g.,  in  two  dimensions,  it  is  a  Kanal  mesh  [l]  [2]) 

3)  is  n-th  order  Markov 

4)  is  weakly  (second  order)  stationary 

5)  is  strongly  stationary 
and  so  on. 

Making  one  of  these  assumptions  means  that  in  reality  we  are  not  consider¬ 
ing  the  problem  of  finding  the  field  that  generated  the  given  sample,  but  some 
other  field  that  belongs  to  the  given  class  and  approximates  the  field  that  gen¬ 
erated  the  given  sample.  In  this  paper  we  will  not  be  interested  in  the  problem 
of  evaluating  how  good  this  approximation  is,  because  this  aspect  is  treated  in 
the  author’s  papers  [8]  [9]. 
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In  this  series  of  papers  we  will  study  the  problem  in  the  case  where  the  field 
is  assumed  to  be  strongly  stationary,  with  some  additional  restrictions. 

1.2.  Direction  of  the  research 

The  first  part  of  our  research,  presented  in  this  paper,  is  concerned  with  the 
simplest  case,  where  we  have  a  stationary  field  made  up  of  independent  random 
variables;  obviously,  we  may  suppose  that  the  field  is  one-dimensional. 

The  next  stage  of  the  research  will  consider  the  case  of  one-dimensional  sim¬ 
ple  homogeneous  Markov  chains,  followed  by  one-dimensional  Markov  homogene¬ 
ous  chains  of  higher  order.  Subsequent  stages  will  study  two-  (or  higher-)  dimen¬ 
sional  Markov  random  fields  (Kanal  meshes),  simple  or  of  higher  orders. 

1.3.  Digitization 

In  order  to  be  able  to  deal  with  digitized  data  and  at  the  same  time  to 
reduce  the  complexity  of  the  problem,  we  will  consider  only  random  fields  with  a 
finite  set  of  possible  outcomes  at  each  point.  In  order  to  extend  these  results  to 
the  continuous  case,  we  would  have  to  consider  some  process  of  approximation, 
such  as  that  used  by  the  author  [3]— [7], 

2.  The  direct  theorem 

2.1.  Generalities 

Let  us  consider  a  sequence  of  independent  trials  with  possible  outcomes 
Ai  (1  <  i  <n)  and  corresponding  probabilities  p,  >  0(1  <  i  <  n)  adding  up  to 
1.  Each  possible  result  of  a  series  of  s  consecutive  trials  can  be  written  as  a 
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sequence 

C,  =  (Akl,A^ . Ak)  (2.1) 

where  each  kT  (1  <  r  <  s)  can  take  any  value  i  (1  <  »  <  n).  Because  of  sta- 
tionarity,  the  probability  of  occurrence  of  the  sequence  C,  does  not  depend  on  the 
moment  when  the  trials  begin;  taking  into  consideration  the  independence  of  the 
trials,  this  probability  can  be  written  as 

pra-np(4)  (2.2) 

Let  us  denote  by  m,  (1  <  i  <  n)  the  number  of  times  the  outcome  .4,  appears  in 
the  sequence  Ca,  so  that 

E  mt  =  s  (2.3) 

i=i 

The  equality  (2.2)  can  be  written 

P(Ct)=flP?'  (2.4) 

i=i 


In  what  follows  we  denote  by 


tt  =  E  Pi  *°g  ~ 
,=1  p, 


(2.5) 


the  entropy  of  the  random  field  characterized  by  the  probabilities  p,  (l<  i  <  n), 
and 


n  1 

P  =  E  lo5  — 
.=  ,  Pi 


(2.6) 
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Obviously 


0  <  p  <  oo 


(2.7) 


2.2.  The  theorem 

Let  us  denote  by  T4  the  class  of  all  sequences  C3.  For  given  6  >  0,  s  >  0  we 
denote  by  the  set  of  all  sequences  Ca  6  such  that 

| mt  -  sp,|  <  s6  (2.8) 

for  all  i  (1  <  i  <  n),  and  by  T*3  its  complement  with  respect  to 

Definition.  Sequences  C3  €  Tj,  will  be  called  (6,s)-standard  sequences  or  simple 
standard  sequences. 


Let  us  consider  the  equation 


and  let  us  denote  by  u  (e)  its  solution. 

Definition.  Given  e  >  0,  <5  >  0,  s  >  n,  condition  A  holds  if 

4  S2  (  s  >  n  (2.10) 

and  condition  B  holds  if 

4  £r  s  >  u2  (f)  (2.1 1 ) 

Let  us  denote  by  .V  (  )  the  cardinality  of  a  set. 
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Theorem  1. 


Let  us  suppose  that  at  least  one  of  the  conditions  A,  B  holds.  Then 

(a)  If  Ct  is  a  (<5,s)-standard  sequence,  it  follows  that 

1  log  TTrT  ~  H  <  6p 

s  P{Ca) 

(b) P(ri.J  >  1  -  e 

(c)  lim  -log  N{VS')  =  H 

i  —  CO  s 
S  -  0 

Remark  1. 

The  relation  (2.12)  is  equivalent  to 

2 -»(H*6p)  p(Ca)  <  2~*  ~  6 

i.e.  to 

P{CS)  =  2~>h+  stpe  ,  \6\  <  1 

Remark  2. 

The  relation  (2.13)  is  equivalent  to 

Pm  <  < 

Remark  3. 


From  (2.14)  it  follows  that 


*v  (p  j 


=  0 


lim 

3  —  00 


A'(rj 


Indeed,  from  (2.14)  we  obtain  the  relation 
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log  N( =  5  •  [H+  o(l]) 

i.e., 


(2.19) 


N(r'6a)  =  2*1 H+  ^  (2.20) 

Taking  into  consideration  that 

N  (f4)  =  nJ  =  24 108  n  (2.21) 


and  because 


H  <  log  n  , 


(2.22) 


if  follows  that 


iV  =  9-i  (log  n-  H+  0(1))  =  0(,\ 

N(r<) 


which  is  equivalent  to  the  first  equality  in  (2.18),  and 

JV(r&)  n  (rj-A*  (n ,)  N(ru 
N(rt)  n  (rj  "  iV(rf) 


(2.23) 


(2.24) 


which  is  equivalent  to  the  second  equality  in  (2.18). 

Remark  4. 

Our  Theorem  1  is  closely  related  to  some  results  which  go  back  to  Shannon 
[10]  and  received  a  mathematically  acceptable  form  from  Khinchine  [2]. 

Our  Theorem  1(a),  (b)  refers  to  independent  random  variables,  while  that  in 
[2]  refers  to  ergodic  simple  Markov  chains,  but  our  result  is  not  a  particular  case 
of  that  in  [2].  Indeed,  the  results  in  [2]  are  existence  theorems,  considering  that  6 , 
e  can  be  taken  as  small  and  .s  as  large  as  desired,  while  our  results  give  effective 
relations  between  <5.  e,  s  in  order  that  the  results  hold. 
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Our  Theorem  1(c)  refers  to  the  set  T/,,  of  all  standard  sequences  Ca,  while 
the  result  in  ([2],  Th.  3)  refers  to  another  set  of  sequences  C3 ;  our  result  contains 
a  limit  for  <5  — »  0,  s  — ►  oo,  while  the  result  in  ([2],  Th.  3)  contains  a  limit  for 
s  — *■  oo. 


2.3.  Proof 


(a)  Let  us  consider  a  sequence  Ca  6  From  (2.8)  it  follows  that 


m,-  =  sp,  +  s66(  |0,|  <1  (1  <  i  <  n) 


(2.25) 


From  (2.4)  there  follows  the  relation 


log  P{Ca)—  £  m,  log  p, 

i=i 


(2.26) 


and  taking  into  consideration  (2.25),  there  follows  the  equality 


log  P(Ca)=  £  (*Pi  +  sM,)  log  p, 

i=i 

n  n 

=  «  £  P,  log  Pi  +  s6  •  2  9,  log  p, 

i=i  i=i 


(2.27) 


which  can  also  be  written  as 


log  p  jfrr  =  sH+s6  £  9i  'og  -7 
r  \  Vs)  i=  1  Pi 


(2.28) 


From  (2.28)  we  obtain  the  result  (a): 


7  '°*TTcTm 


<  6  ■  5]  I  9i  I  log  —  <  6  •  £  log  —  =  bp  (2.29) 
1=1  Pi  1=1  Pi 
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(b)  Instead  of  proving  inequality  (2.13)  we  will  prove  (2.17).  In  order  that  a 
sequence  Ca  €  r,  belong  to  r"4,  it  is  necessary  that  for  at  least  some  value 
of  I  (1  <  «'  <  n)  the  inequality  (2.8)  does  not  hold,  i.e., 


=  U  i  lm<"  sp'l  >  s6 

i=i 


(2.30) 


so  that 


P  (r  l, )  =  jj  jlm.-sp.l  >  J  j  <  £  /"Jim,  -  sp,\  >  s6  j  (2.31) 

(bl)  Let  us  assume  that  condition  A  holds.  It  is  known  from  the  elements  of  the 
Theory  of  probability  that 


P 1 1  m,  -  sp,\  >  s6 1  < 


Pi  (I-P.) 
sS2 


(2.32) 


But  for  0  <  x  <  1,  we  have  the  inequalities 


0<z(l-.r)<-— 
“  1  '  ~  A 


(2.33) 


where  the  maximum  value  is  reached  for  x  =  — .  so  that  from  (2.32)  it  fol¬ 


lows  that 


pj  |  m,  -  sp,|  >  s<5  j  <  (1  <  t  <  n) 


(2.3-1) 


Consequently,  from  (2.31)  there  follows  the  inequality 


and  because  of  (2.10).  it  follows  that  (2.17)  holds. 


’.35) 
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(b2)  Let  us  assume  that  condition  B  holds.  From  the  Central  limit  theorem 


the  Moivre-Laplace  form,  it  is  known  that 


—  f 

\Z2tt  Jq 


p.  (i  -  p.) 

e  2  dx  (1  <  i  <  n) 


so  that 


P{K-^ 


p.  (i  -  p.)  _£_ 


e  *  dx  ,  ( 1  <  i  <  n) 


In  order  to  obtain  the  relation  (2.17)  it  is  sufficient  to  take 


p,n  -/>,) 

e  2  d x  <  —  (1  <  j  <  n) 

n 


i  f 

\/2n  dQ 


V  p.d  -  p.)  _i_ 


e  2  dx  >  i-  l  -  —  (1  <  i  <  n) 

2  n 


which  is  equivalent  to  the  inequality 


by  p,  (1-p,)  >  u(£)  0  <  <  <  n) 


Because  of  (2.33),  we  have  the  inequality 


(l-p,)  >  2 b\fs  (!</<«) 


iVl'kJ 


so  that  in  order  to  satisfy  (2.-10)  it  is  sufficient  to  lake  in  consideration  Con¬ 
dition  B  (2.11),  i.e. 

26\Ts  >  u  (c)  (2.-42) 

(c)  If  C3  6  F/  3,  then  (2.15)  holds,  so  that 

n (r(.)  2~3^h+  w  <  £  P(ct)  =  P(r|,)  <  1  (2.-13) 

where  the  summation  is  for  all  C,  G  r|s.  From  (2.-43)  there  follows  the  rela¬ 

tion 

-  •  log  jV(r/,)  <  H+  6p  (2.-44) 

S 

In  a  similar  way,  from  (2.13),  (2.15)  there  follow  the  relations 

1  -  6  <  P  (r|  J  =  £P(C3)  <  N(r'Sl)  o-s(ff-Sp)  (2.45) 

where  the  summation  is  also  for  all  Cs  €  r^5.  From  (2.45)  we  obtain  the 

relation 

H-6p<  -  log  N  (r|J  +  1  log  —5—  (2.46) 

s  s  1  -  e 

From  (2.44),  (2.46)  it  follows  that 

H-  6p  -  —  log  — - —  <  —  log  A’(r;3)  <  H  +  bp  (2.47) 

S  1  -  €  S 

For  e  given,  arbitrary,  6  as  small  as  we  want,  and  s  as  large  as  we  want, 
because  of  (2.7)  it  follows  that  (2.14)  holds. 
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3.  The  inverse  theorem 


3.1.  Generalities 


Let  6  >  0,  e  >  0,  s  >  1,  and  let  be  an  arbitrary  specific  sequence. 


belonging  to  T4.  Let  us  assume  that  one  of  the  conditions  A  or  B  holds. 


In  what  follows  we  assume  that  is  generated  by  a  sequence  of  indepen¬ 


dent  trials,  with  possible  outcomes  A,  (1  <  t  <  n)  with  unknown  probabilities 


Pi  (1  <  i  <  n),  and  we  will  try  to  determine  some  intervals  in  which  these  proba¬ 


bilities  can  take  values.  Let  us  denote 


=  mi  (C?)  (1  <  *  <  ") 


and  by  W{5}  the  confidence  of  statement  5. 


3.2.  The  theorem 


Because  we  have  proved  that 


P<n,)>  l-e,P(T^)<  e 


it  follows  that  with  confidence  larger  than  1  -  e,  G  f/4,  i.e.. 


|m°  -  sp,|  <  Ss  ,  (1  <  i  <  n)  |  >  1  -  « 


(  m?  m?  I 

A  — -  -  <5  <  Pi  <  — -  +  <5  ,  (1  <  *  <  n)  >  1  - 

Vs  s  1 


Let  Ln  be  the  Banach  space  of  all  vectors 


-  ",  ’•  ■  s *'  *  -  -  ^  -  ■"  -  ^ 


with  g,  real  numbers  of  any  sign,  with  norm 

ll?ll  =  sup  j  |g,|;  1  <  »  <  n  j 

Let  nn  be  the  totality  of  probability  measures 

P  =  (Pi.  •  •  Pn) 

with  p,  >  0  (1  <  »  <  n),  and 

E  Pi  =  1 

1=1 


(3.6) 


(3.7) 


(3.8) 


This  is  a  metric  space  with  distance 

Up  -  P'11  =  sup  J  |p,  -  p\  I;  1  <  »  <  n  J  (3.9) 

where  p,  p'  6  nn  p  -  p'  €  Ln.  If  p,  p'  €  n„  are  two  different  solutions,  satisfying 
the  inequalities  in  (3.4),  it  follows  that 

|p, -p: I  <25  (l<i<»)  (3.10) 

so  that  from  (3.9)  it  follows  that 


Up  -  p'l!  <  25 


(3.11) 


We  have  thus  proved 

Theorem  2. 


Let  us  assume  that 


(1)  e,  6,  s  satisfy  one  of  the  conditions  A,  B; 

(2)  the  arbitrary  sequence  C$  6  Ta  is  generated  by  an  independent  identically 
distributed  sequence  of  trials,  with  unknown  probabilities  p,  (1  <  :  <  n). 

Then 

(a)  The  relation  (3.4)  holds. 

(b)  If  p,  pi  are  two  different  solutions,  their  distance  in  FIn  is  less  than  26. 

Remark  4. 

Let  be  the  Banach  space  of  all  vectors  (3.5)  with  norm  the  total  variation 

llklil  =  S  kl  (3.12) 

i=i 

Then  n„  is  a  metric  space  with  distance 

III?  -  P'111  =  t  IPi-Pjl  (3.13) 

i=i 

where  p,  p'  €  nn. 

If  P,  pl  6  nn  are  two  different  solutions,  satisfying  (3.4),  it  follows  from  (3.13) 
that 

IIIp  -  pill  <  2n6  (3.14) 


It  is  easy  to  see  that 

Up  -  P'11  <  IIIp  -  P'111  <  ?iIIp  -  P'11  (3-i5) 

We  remark  also  that  if  L "  is  the  Euclidean  space  of  all  vectors  (3.5)  with 


norm 


2 


(3.16) 


((«))-(£  9?| 

1  i=i  t 

then  nn  is  a  Euclidean  space  with  distance 

«p -✓))-(£ Ip,-p!I2)2  (3.17) 

It  is  easy  to  see  that 

Up-  P'11  <  ((p-  p'))  <  n/«IIp -  P'11  (3.18) 

4.  Examples 

4.1.  Examples  under  Condition  A 

Example  1. 

Let  Cj  be  a  sequence  with  n  —  2,  s  —  104,  e  =  2~3  —  0.125,  6  >  0.02,  so 
that  condition  A  holds.  Let  m,  =  3  X  103,  m2  =  7  X  103. 

From  (3.4)  it  follows  that 

ivj  0.28  <  p,  <  0.32;  0.68  <  p2  <  0.72  j  >  0.875  (4.1) 

and  from  (3.11)  we  obtain 

||p  -  p'H  <  0.04  (4.2) 

Example  2- 


Let  C®  be  a  sequence  with  n  =  2,  s  —  106,  e  =  2' 3  =  0.125,  6  >  0.002.  so 
that  condition  A  holds.  Let  m°  =  3  X  105,  m£  =  7  X  105. 


a  *  t/U*  lUtl . 1 


From  (3.4)  it  follows  that 


'|  0.298  <  ?!  <  0.302;  0.698  <  p2  <  0.702  j  >  0.87c 


and  from  (3.11)  we  obtain 


||p-  p'i)  <  0.004 


4.2.  Examples  under  Condition  B 


Example  3- 


Let  C®  be  a  sequence  with  n  =  2,  e  =  2"3  =  0.125,  $  =  lO4,  6  >  0.009, 
=  3  X  103,  mS  =  7  X  103,  so  that 


—  1  -  -  =  0.46875 
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and  relation  (2.39)  takes  the  form 


1=/ 
2t r  J0 


v  (e)  -4 

e  *  dx  >  0.468/5 


which  holds  for 


u  (e)  >  1.8 


Considering  Condition  B  in  form  (1.42)  it  is  easy  to  see  that  it  holds.  From 
(3.4)  it  follows  that 

W |  0.291  <  Pi  <  0.309  ;  0.691  <  p2  <  0.709  j  >  0.875  (4.8) 


and  from  (3.11)  it  follows  that 


Up -P'1!  <0.018 


(•*•8) 


Example.  4. 

Let  C°a  be  a  sequence  with  n  =  2,  f  =  2~'3  =  0.125,  s  =  106,  6  >  0.0009, 
=  3  X  105,  m2  =  7  X  105;  in  this  case,  relations  (4 .5)— (-4 .8)  hold,  so  that 
Condition  B  holds.  From  (3.4)  it  follows  that 


VFj  0.2991  <  px  <  0.3009  ;  0.6991  <  p2  <  0.7009  j  >  0.875  (4.10) 

and  from  (3.11)  it  follows  that 

||p  -  P'11  <  0.0018  (4.11) 

4.3.  Examples  involving  images  that  satisfy  Condition  A  or  B 


Example.  ,5- 

Let  us  consider  a  digital  television  picture,  i.e..  an  array  of  5002  points, 
where  each  point  can  have  256  levels  of  gray. 

Here  n  =  256,  s  =  5002  =  250,000;  let  e  =  — =  0.00390625. 

2o6 

Taking  these  values,  if  we  want  Condition  A  satisfied  it  is  sufficient  that 

45s  X  250,000  X  >  256  (4.12) 

mOO 

or 


i.e. 


106  &  >  2562  , 


(4.13) 


<5  >  0.256 


(4.14) 
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Consequently 


I  m?  1 

- p,  <  0.256  ;  (1  <  »  <  256)  j  >  0. 


9960937  (-4-15) 


|p  -  p'll  =  max  |  |p,  -  pj 1  ,  1  <  »  <  256  J  <  0.512 


(4.16) 


With  the  same  basic  data  as  in  Example  5,  we  take  n  =  256,  s  =  5002, 

t  =  — =  0.00390625,  and  we  consider  that  Condition  B  holds,  i.e., 

256 


26\/s  >  u  (f) 


(4.17) 


2  n  2  2562  -  65.536 


—  1 - - -  ~  -  (1  -  0.16667)  =  -  X  0.83334 

2  60,000  2  2 


=  0.41667 


so  that  from  tables  it  follows  that 


u  {()  ~  1.30 


Thus 


(4.18) 


(4.19) 


2(5  X  500  >  1.30 


(4.20) 


6  >  0.0013 


(4-21) 
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J 


■-  ■-  *.  \  . 


— - P,  <  0.0013  ;  (l  <  I  <  256))  >  0.9060037 


||p-  pll  <  o.oo26 


(  1  231 


Let  us  take  n  —  256,  s  —  500",  t  —  — —  =  0.0625.  and  let  us  assume  that 

1 6 

Condition  A  holds.  Then 


46s  X  250,000  X  —  >  256 
16 


106  &  >  212 


6  >  0.064 


(4.24) 


(4.25) 


(4.26) 


so  that 


w\  —  -  Pt  <  0-064  ;  1  <  »  <  256  I  >  0.0375  (4.27) 


||p-  P'11  <  o.i28 


(4.28) 


tmple  8. 


Let  n  —  256,  s  —  250,000,  e  =  =  0.0625  and  let  us  assume  that  Condi- 

16 

tion  B  holds.  Then 


K 


1  1-i. 

O  n 


1  i  -  -L  _L 

2  16  256 


—  f  i - — 

2  4096 


1  1 - - — 

2  4000 


—  -i-  (1  -  0.00025)  =  —  X  0.99975  =  0.49987 

2 


(4.29) 


so  that 


u  (<)  ~  3.8 


26  X  500  >  3.8 


6  >  0.0038 


f  m°  I 

W\  —  -  Pi  <  0.0038  ;  1  <  t  <  256  j  >  0.9375 


||p  -  p'H  <  0.0076 


(4.30) 


(4.31) 


(4.32) 


(4.33) 


(4.34) 


Let  us  assume  that  we  have  a  30-minute  sequence  of  TV  pictures.  If  we  have 
32  pictures  in  each  second,  we  have  a  total  of 


32  X  60  X  30  =  24  X  602 


pictures,  succeeding  each  other  in  time.  Assuming  independence  between 
the  pictures,  we  have  n  ~  256,  s  =  500*  X  24  X  602,  and  let 

e  —  =  0.00390625.  Assuming  that  Condition  A  holds,  the  value  of  6 


is  given  by 


4 S2  (250.000)  X  24  X  602  X  -J-  >  256 


(4.36) 


WTlTlTlTt 


106  F  X  24  X  602  >  256" 


(4.37) 


Then 


103  6  X  22  X  60  >  256 


°56 

A  >  — -= -  >  0.001 

102  X  240 


Consequently 


(4.38) 


(4.39) 


(  m?  I 

W'j - p,  <  0.001  ;  (1  <  1  <  256)  j  >  0.9960937 


(4.40) 


||p-  P'11  <  0.002 


(4.41) 


so  that 


Q-Ek-IQ. 


Let  us  consider  the  same  problem  as  in  Example  9.  with  the  supposition  that 
Condition  B  holds. 

In  this  case 


2  A  (.500  X  22  X  60)  >  1.30 


A  >  - - - -  0.0000054 

2.400.000 


(4.42) 


H'j  —  p,  <  0.0000054  :  (1  <  i  <  256)  j  >  0.9960937  (4.44) 


ip  p'!|  <  o.ooooio* 
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